Agilis: An Internet-Scale Distributed Event Processing System for Collaborative Detection of Cyber Attacks

As cyber attacks become increasingly distributed and sophisticated, so must our defenses. Collaborative processing of data produced by independent sources is advantageous for early and accurate detection of Internet-based threats, and instrumental for identifying complex attack patterns that target multiple administratively and geographically disjoint entities. In this paper, we introduce Agilis – a lightweight collaborative event processing platform for sharing and correlating event data generated in real time by multiple widely distributed sources. The primary goal of the Agilis design is to tread the balance between simplicity of use, robustness and scalability on one hand, and reasonable performance in large-scale settings on the other. To this end, Agilis is built upon the open-source Hadoop’s MapReduce infrastructure, which we augmented with a RAM-based data store and various locality-oriented optimizations to improve responsiveness and reduce overheads. The processing logic is specified in a flexible high-level language, called Jaql, which supports data flows and SQL-like query constructs. We demonstrate the utility of the Agilis framework by showing how it facilitates the collaborative detection of two different exploits: stealthy inter-domain port scans used by hackers for reconnaissance, and a botnet-driven HTTP session hijacking attack. We evaluate the performance of Agilis in both scenarios, and, in the case of interdomain port scans, compare it to a centralized high-end event processing system called ESPER. Our results show that while Agilis is slower than ESPER in a local area network, its relative performance improves substantially as we move towards larger scale distributed deployments.